Skip to content

refactor(index): pass resolved bucket mode to index manifests - #494

Merged
SteNicholas merged 3 commits into
apache:mainfrom
fastio:fix/index-manifest-resolved-bucket-mode
Nov 3, 2026
Merged

SteNicholas merged 3 commits into
apache:mainfrom
fastio:fix/index-manifest-resolved-bucket-mode

Conversation

@fastio

@fastio fastio commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: part of #493

IndexManifestFile and IndexManifestFileHandler received the configured bucket option and
treated -1 as BUCKET_UNAWARE when choosing how to combine deletion vector index entries. A
primary-key table with dynamic buckets also has bucket = -1, but keeps one deletion vector index
per (partition, bucket), so it must use the bucketed combiner, as Java does for
BucketMode.HASH_DYNAMIC.

This change passes the resolved BucketMode (ResolveBucketMode(options.GetBucket(), schema))
everywhere an IndexManifestFile is created, and only uses the global deletion vector combiner
for BUCKET_UNAWARE. That combiner merges entries by index file name and uses data file names
only to reject a data file covered by more than one deletion vector.

There is no behavior change for tables that can be written today: dynamic bucket primary-key
tables are still rejected by the writer. This is the first step of #493; batch reads of those
tables follow in a separate PR.

Tests

  • IndexManifestFileHandlerTest: existing cases now pass an explicit BucketMode; added
    DynamicBucketDvUsesBucketedCombiner, which goes through IndexManifestFile::Create() and
    WriteIndexFiles() so the stored bucket mode is exercised end to end. It checks that a new
    deletion vector index of a HASH_DYNAMIC table replaces the previous one of the same
    (partition, bucket) and leaves other buckets untouched, while a separately created
    BUCKET_UNAWARE manifest rejects the same delta with
    Trying to add a second deletion vector for data file data-0.orc.
  • IndexFileHandlerTest resolves the bucket mode with ResolveBucketMode() from the table
    schema, the same way production callers do.
  • Updated callers in CommitScannerTest, ExpireSnapshotsTest, FileSystemWriteRestoreTest
    and OrphanFilesCleanerTest.

Verified on Ubuntu 24.04 (aarch64), GCC 13, Debug: the affected suites pass (40 tests) and the
full paimon-core-test passes (2536 tests).

API and Format

No public API change (only internal headers under src/). No storage format or protocol change.

Documentation

No.

Generative AI tooling

Generated-by: Claude Code (Claude Opus 5.5)

@SteNicholas SteNicholas left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fastio, thanks for contribution. I left some comments for this pull request. PTAL.

@lxy-9602

lxy-9602 commented Nov 3, 2026

Copy link
Copy Markdown
Member

Hi, @fastio I saw the #493 issue—thank you for your contribution and the submission, and also thanks to @SteNicholas for the review. Aside from the review mentioned above, I don’t have any other concerns. Once the fix is complete, please ping us so we can merge.

@fastio

fastio commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@SteNicholas @lxy-9602 All comments are addressed in eb2b040, PTAL.

@SteNicholas SteNicholas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for contribution.

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@SteNicholas
SteNicholas merged commit 42154fc into apache:main Nov 3, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants